Perception techniques are applied to sensor data collected for an environment in order to understand what objects are present in the environment, where the objects are located in the environment, and how these objects move or change within the environment. Because multiple sensors are often employed to collect sensor data in different modalities, sensor data may offer a rich data set to analyze when performing perception tasks. Therefore, the collected sensor data may produce different perception decisions dependent upon the type of sensor data that is evaluated. Consumers of perception decisions, such as control systems that respond to changes in the environment by directing various control actions of system resources, may leverage the multiplicity of perception decisions to determine the control actions that are performed.